System and method for generating enhanced stereographic videos of aircraft build processes

ABSTRACT

Provided is a system and method for generating enhanced stereographic videos of aircraft build processes. Specifically, the system comprises a stereoscopic recording device configured to capture a plurality of stages of an aircraft build process. The system further comprises one or more processors, memory, and one or more programs stored in the memory that comprise instructions for execution by the system to build a stereographic library including repositories of 3D video corresponding to the plurality of stages of the aircraft build process. The system then generates an enhanced walkthrough video of the aircraft build process. The enhanced walkthrough video may include a parallax grid overlay and/or a thermal scan overlay integrated into the video. The system may then analyze the enhanced walkthrough video using post-processing analytics to identify anomalies and irregularities that occurred during the aircraft build process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/600,271, entitled: “SYSTEM AND METHOD FOR GENERATING ENHANCEDSTEREOGRAPHIC VIDEOS OF AIRCRAFT BUILD PROCESSES” filed on Oct. 11,2019, which is a continuation of U.S. patent application Ser. No.15/209,733, entitled: “SYSTEM AND METHOD FOR GENERATING ENHANCEDSTEREOGRAPHIC VIDEOS OF AIRCRAFT BUILD PROCESSES” filed on Jul. 13,2016, now issued as U.S. Pat. No. 10,445,867 on Oct. 15, 2019. Bothapplications are incorporated herein by reference in their entiretiesfor all purposes.

BACKGROUND

During the manufacturing process of an aircraft, it is difficult forcustomers to gain an understanding of the progress of their productbeing built. Currently, customers must physically travel to the factoryin order to review the manufacturing progress of an aircraft or otherlarge-scale product with high costs. This may involve a travel itinerarythat includes significant costs (e.g. airfare, hotel, car rental, etc.)leading to a brief physical walkthrough of the final product, usually inthe final aircraft acceptance position. If multiple reviews are desired,the process must be repeated adding additional cost and time for thecustomer. Companies have attempted to utilize virtual showrooms of thecomputer aided design (CAD) data. However, virtual showrooms onlypresent the expected end-state of the aircraft and not an actual view ofthe aircraft at different stages of the build process. Thus, there is aneed for an improved system and method for supporting inspection of anaircraft at different stages of the build process that is morecost-effective and time-efficient than current methods.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding of certain embodiments of thisdisclosure. This summary is not an extensive overview of the disclosure,and it does not identify key and critical elements of the presentdisclosure or delineate the scope of the present disclosure. Its solepurpose is to present some concepts disclosed herein in a simplifiedform as a prelude to the more detailed description that is presentedlater.

Provided are various systems and methods for generating enhancedstereographic videos of aircraft build processes. In variousembodiments, a system comprises a stereoscopic recording deviceconfigured to capture a plurality of stages of an aircraft buildprocess. The system further comprises one or more processors, memory,and one or more programs stored in the memory. The one or more programsmay comprise instructions for building a stereographic library includingrepositories of 3D video organized by tail number. The repositories of3D video may correspond to the plurality of stages of the aircraft buildprocess.

The one or more programs further comprise instructions for generating anenhanced walkthrough video of the aircraft build process. The enhancedwalkthrough video may include one or more of the following: a parallaxgrid overlay integrated into the video, and a thermal scan overlayintegrated into the video. The parallax grid may include a plurality ofparallax lines determined automatically using autofocus. The pluralityof parallax lines may be organized as one or more sets of parallaxlines. Each set of parallax lines may be stored as a separate videolayer in the parallax grid overlay. The parallax grid may be configuredsuch that accurate real-life measurements for locations, spacing, andaircraft structures can be extracted from the enhanced walkthroughvideo. The system may be configured to provide remote in-process initialinspections capabilities.

The one or more programs further comprise instructions for analyzing theenhanced walkthrough video using post-processing analytics to identifyanomalies and irregularities that occurred during the aircraft buildprocess. In some embodiments, the post-processing analytics includesanalyzing patterns and shapes to detect foreign object damage. In someembodiments, the post-processing analytics includes analyzing patternsand shapes to determine assembly and sub-assembly compliance. In someembodiments, the post-processing analytics includes analyzing patternsand shapes to determine thermal gradient compliance.

Provided also is a method for generating enhanced stereographic videosof aircraft build processes. According to various embodiments, themethod comprises capturing a plurality of stages of an aircraft buildprocess via a stereoscopic recording device. The method furthercomprises building a stereographic library including 3D repositories ofvideo organized by tail number. The 3D repositories of video maycorrespond to the plurality of stages of the aircraft build process. Themethod further comprises generating an enhanced walkthrough video of theaircraft build process, the enhanced walkthrough video including one ormore of the following: a parallax grid overlay integrated into thevideo, and a thermal scan overlay integrated into the video. Theenhanced walkthrough video may be configured to allow remote in-processinitial inspections. The method further comprises analyzing the enhancedwalkthrough video using post-processing analytics to identify anomaliesand irregularities that occurred during the aircraft build process.

Other implementations of this disclosure include corresponding devices,systems, and computer programs, configured to perform the actions of thedescribed method. For instance, a non-transitory computer readablemedium is provided comprising one or more programs configured forexecution by a computer system. In some embodiments, the one or moreprograms include instructions for performing the actions of describedmethods and systems. These other implementations may each optionallyinclude one or more of the aforementioned features. These and otherembodiments are described further below with reference to the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of an example network architecture forimplementing various systems and methods of the present disclosure, inaccordance with one or more embodiments.

FIG. 2 is a schematic illustration of a stereoscopic camera that can beused in conjunction with the techniques and mechanisms of the presentdisclosure.

FIGS. 3A and 3B illustrate an example of an image of the interior of anaircraft and a panoscan of the image, in accordance with someembodiments.

FIG. 4A illustrates an example of an image displaying the interior of anaircraft, in accordance with some embodiments.

FIGS. 4B-4F illustrate examples of an image displaying the interior ofan aircraft with an integrated overlay, in accordance with someembodiments.

FIGS. 5A-5B illustrate a process flowchart corresponding to a method forgenerating enhanced stereographic video of aircraft build processes, inaccordance with some embodiments.

FIG. 6 is a block diagram illustrating an example of a computer systemcapable of implementing various processes described in the presentdisclosure.

FIG. 7 is a schematic illustration of an aircraft, in accordance withsome embodiments.

FIG. 8 is a block diagram of aircraft production and service methodologythat may utilize methods and assemblies described herein.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the presented concepts. Thepresented concepts may be practiced without some or all of thesespecific details. In other instances, well known process operations havenot been described in detail so as to not unnecessarily obscure thedescribed concepts. While some concepts will be described in conjunctionwith the specific embodiments, it will be understood that theseembodiments are not intended to be limiting. On the contrary, it isintended to cover alternatives, modifications, and equivalents as may beincluded within the spirit and scope of the present disclosure asdefined by the appended claims.

For example, the techniques of the present disclosure will be describedin the context of particular systems used for aircraft fabrication.However, it should be noted that the techniques and mechanisms of thepresent disclosure apply to generating enhanced videos for differentbuild processes in various other industries. In the followingdescription, numerous specific details are set forth in order to providea thorough understanding of the present disclosure. Particular exampleembodiments of the present disclosure may be implemented without some orall of these specific details. In other instances, well known processoperations have not been described in detail in order not tounnecessarily obscure the present disclosure. Various techniques andmechanisms of the present disclosure will sometimes be described insingular form for clarity. However, it should be noted that someembodiments include multiple iterations of a technique or multipleinstantiations of a mechanism unless noted otherwise.

Various techniques and mechanisms of the present disclosure willsometimes be described in singular form for clarity. However, it shouldbe noted that some embodiments include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. For example, a system uses a processor in a variety ofcontexts. However, it will be appreciated that a system can use multipleprocessors while remaining within the scope of the present disclosureunless otherwise noted. Furthermore, the techniques and mechanisms ofthe present disclosure will sometimes describe a connection between twoentities. It should be noted that a connection between two entities doesnot necessarily mean a direct, unimpeded connection, as a variety ofother entities may reside between the two entities. For example, aprocessor may be connected to memory, but it will be appreciated that avariety of bridges and controllers may reside between the processor andmemory. Consequently, a connection does not necessarily mean a direct,unimpeded connection unless otherwise noted.

Overview

The present disclosure describes a novel system and method for providingimproved virtual inspection of aircraft build processes. The systemprovides remote access for customers to view the periodic progress ofthe manufacture of an assembly product by remotely accessing video dataacquired during the assembly with a recording device. Specifically, thesystem may include building a stereographic library includingrepositories of 2D and/or 3D images, including video, captured by arecording device, such as a stereoscopic camera. The recording devicemay be wearable by an operator such that video data is obtained while anoperator walks through the assembly and provides a view of portions ofthe assembly of interest to a customer. The repositories of images maycorrespond to one or more stages of the aircraft build process and maybe organized by aircraft tail number. The 2D and/or 3D images may bepanoramic images with true perspective that can provide up to a 360degree view of the interior and exterior of the aircraft and a morerealistic and comprehensive view of a particular build stage as comparedto current panoscans and other methods.

In some embodiments, parallax lines, including station-lines,butt-lines, and water-lines, may be generated for each captured image bya laser projection device equipped with autofocus capabilities. Theparallax lines may be organized as sets of one or more grouped parallaxlines, and each set of parallax lines may be stored as an overlay. Theparallax lines may be evenly spaced at a predetermined distance (e.g.,5″, 10″, 12″, 18″, etc.) and provide scaling and perspective for theaircraft structure in horizontal, vertical, and longitudinal directions.In addition, the system may capture other information relevant to eachbuild stage, such as foreign object damage, assembly compliance, andthermal deviation. This additional information may be stored in variousoverlays either separately and/or in combination. The one or moreoverlays may be activated (e.g., turned on/off) as needed to supportengineering analysis and/or customer inspection. Post-processinganalytics may be employed to automatically compare the captured imagesagainst database information to detect foreign object damage, determineassembly and sub-assembly compliance, and determine thermal gradientcompliance.

The improved system has many advantages in comparison with conventionalsystems and methods for providing inspection of aircraft structures. Forexample, the disclosed system creates a library of videos and images ofmultiple build stages that may be conveniently accessed on demand fromany location for virtual inspection of an aircraft structure. Thiseliminates the need for costly and time consuming travel to the physicallocation of the aircraft. Additionally, the system can provideundistorted 3D images with a larger field of view for more accurateperspective compared to traditional panoscans. One having ordinary skillin the art would recognize that current use of panoscans presentdistortions and limited range of view making it difficult to visualizeareas outside of the view of reference and to scale the structures shownin the image.

Furthermore, parallax line overlays provide accurate measurements forlocations, spacing, and aircraft structures, while thermal scan overlaysallow technicians to safely and quickly identify potential systemmalfunctions. Post-processing analytics may also improve quality controlby automatically identifying foreign object damage (“FOD”), determiningassembly compliance, and/or analyzing thermal deviation. Overall, theimproved system and methods may result in early identification ofmanufacturing issues and deviations from customer requirements resultingin significant cost savings and more accurate builds for bothmanufacturers and customers.

Example Embodiments

FIG. 1 illustrates a diagram of an example network architecture 100 forimplementing various systems and methods of the present disclosure, inaccordance with one or more embodiments. The network architecture 100includes a number of client devices 102-108 communicably connected toserver system 112 by a network 110. In some embodiments, server system112 includes one or more processors and memory. The processors of serversystem 112 execute computer instructions (e.g., network computer programcode) stored in the memory to perform functions of a network dataexchange server.

In some embodiments, server system 112 is a content server configured toreceive and store repositories of video and/or image files recorded byrecording device 118. Server system 112 may also function as a dispatchserver configured to transmit and/or route network data packetsincluding network messages. In some embodiments, the networkarchitecture 100 may further include database 116 communicably connectedto client devices 102-108 and server system 112 via network 110. In someembodiments, network data, stereographic libraries including 3D videorepositories, or other information such as network profile information,aircraft information, manufacturing information, build processinformation, etc., may be stored in and/or retrieved from database 116.In some embodiments, database 116 is a component within server system112 and stored within memory of server system 112.

Users of the client devices 102-108 may access the server system 112 toparticipate in a network data exchange service. For example, the clientdevices 102-108 can execute web browser applications that can be used toaccess the data stored in server system 112 and/or database 116. Inanother example, the client devices 102-108 can execute softwareapplications that are specific to the network (e.g., networking dataexchange “apps” running on smartphones). Users interacting with theclient devices 102-110 can participate in the network data exchangeservice provided by the server system 112 by accessing stored data andreviewing 3D video images and distributing digital content, such as textcomments (e.g., messages inquiries, updates, announcements, replies.

In some implementations, the client devices 102-110 can be computingdevices such as laptop or desktop computers, smartphones, personaldigital assistants, portable media players, tablet computers, or otherappropriate computing devices that can be used to communicate with anelectronic network. In some implementations, the server system 112 caninclude one or more computing devices such as a computer server. In someimplementations, the server system 112 can represent more than onecomputing device working together to perform the actions of a servercomputer (e.g., cloud computing). Network 110 may be a wired and/orwireless network. In some implementations, the network 110 can be apublic communication network (e.g., the Internet, cellular data network,dial up modems over a telephone network) or a private communicationsnetwork (e.g., private LAN, leased lines).

FIG. 2 is a schematic illustration of a stereoscopic camera 200 that canbe used in conjunction with the techniques and mechanisms of the presentdisclosure. Dashed lines within FIG. 2 indicate optional components tostereoscopic camera 200. In some embodiments, stereoscopic camera 200may be recording device 118 previously described in FIG. 1. In variousembodiments, stereoscopic camera 200 includes a body 202 with two ormore lenses, such as lenses 204 a and 204 b. Stereoscopic camera 200 maybe configured to capture 2D and/or 3D video and/or images through stereophotography. Each lens, such as lenses 204 a and 204 b may include aseparate image sensor and/or film frame allowing the camera to simulatehuman binocular vision and to capture 3D images. In some embodiments,stereoscopic camera 200 may be a combination of two or more imagecapturing devices. In some embodiments, stereoscopic camera 200 mayinclude any other suitable lens mechanism for capturing 3D images, suchas a 2D/3D interchangeable prime lens. In some embodiments, recordingdevice 118 may comprise a plurality of cameras mounted such that thecameras capture a 360 degree panoramic view of the surroundings. Such aconfiguration of cameras may be coupled with smart processing softwareto create seamless panoramic displays in 2D and/or 3D. Panoramic videoand/or images captured by stereoscopic camera 200 may provide up to a360 degree view of the surroundings of the interior and/or exterior ofan aircraft. In some embodiments, the panoramic video and/or imagesinclude a true perspective view that provides more accurate depthperspective and less view distortion than current panoscans, such aspansocans taken with a digital panoramic rotating line camera.

The body 202 of stereoscopic camera 200 may be coupled to a mount 208allowing stereoscopic camera 200 to be mounted upon various structures.For example, in some embodiments, stereoscopic camera 200 may be mountedupon a rail system. In some embodiments, the rail system may allowstereoscopic camera 200 to rotate up to 360 degrees in order to capturea larger view of reference. In other embodiments, stereoscopic camera200 may be mounted upon the helmet of an operator and may record videoand/or images of portions of the assembly of interest to a customer asan operator travels along the interior and/or exterior of the aircraftduring each incremental stage of manufacturing. The 3D images capturedby stereoscopic camera 200 may then be formatted for viewing withappropriate 3D viewing devices, such as virtual reality goggles, etc.

In some embodiments, stereoscopic camera 200 may include a deviceattachment 210 mounted to an area of the body 202 of stereoscopic camera200. As depicted in FIG. 2, mounted device 210 is mounted at the top ofstereoscopic camera 200. However, mounted device 210 may be mounted onany portion of stereoscopic camera 200 in different embodiments. In someembodiments, stereoscopic camera 200 may include a plurality of mounteddevices 210 mounted on various portions of body 202. In variousembodiments, mounted device 210 may capture information that enhances orsupplements the 3D image captured by stereoscopic camera 200. In oneaspect, mounted device 210 may be a laser projection device with anautofocus capability for creating referential measurement lines asfurther described below with reference to FIGS. 4B-4E. In anotheraspect, mounted device 210 may comprise an infrared vision device forimaging thermal radiation as further described below with reference toFIG. 4F. Such mounted devices 210 may create synchronized overlays thatprovide additional information regarding the recorded images. In someembodiments, a mounted device 210 may comprise a lighting source toprovide adequate and/or even lighting of the surroundings. Thecapabilities of one or more of the previously described deviceattachments 210 may be incorporated in and performed by stereoscopiccamera 200.

Examples of Captured Images and Overlays

By capturing video and/or images with more accurate visual perspective,systems and methods of the present disclosure can avoid the problems oftraditional panoscans, and other panoramic photography methods,currently used. FIGS. 3A-3B illustrate an example of an image 300 of theinterior of an aircraft and a panoscan 301 of the image 300, inaccordance with some embodiments. With reference to FIG. 3A, image 300depicts the interior of an aircraft during a build stage. With referenceto FIG. 3B, panoscan 301 may be a static panoramic image of the aircraftthat is displayed and distorted in an extreme wide area lens format, andmay be used in identifying potential locations for engineeringequipment. In some embodiments, image 300 and/or panoscan 301 may be aframe in a video sequence. A panoscan, such as panoscan 301, may readilyreveal hydraulic lines, electrical wires, and/or other engineeringstructures. However, the formatting causes distortions in the view(e.g., a fish eye effect), as shown in FIG. 3B. These distortions maymake it difficult to visualize areas outside of the view of referenceand to determine distances the structure shown in the image.Additionally, the view of reference is fixed at the center of thepicture, and only limited movement is capable, such as zoom in, zoomout, and rotation, and such limited movements may cause the image todistort even further. Furthermore, fish-eye effect distortions createsdifficulty in scaling the existing structures within a panoscan image,such as panoscan 301.

In contrast, a video and/or 360 degree image captured by recordingdevice 118 displays an image with true visual perspective, allowingscaling through various means, such as by overlays of lines spacedevenly apart at known distances. Additionally, a 3D video and/or imagewould provide depth perception information not shown in traditionalpanoscan images. Furthermore a video and/or 360 degree image captured byrecording device 118 is not locked into a particular static view pointand would allow visualization of areas outside of the current view ofreference.

The following figures provide additional examples of recorded images,such as those captured by recording device 118, or stereoscopic camera200, and stored in server system 112 and/or database 116. FIG. 4Aillustrates an example of an image 400 displaying the interior of anaircraft, in accordance with some embodiments. As depicted in FIG. 4A,image 400 includes a plurality of rows of seats, overhead compartments,and windows. Image 400 may be a frame of a recorded video. In someembodiments, image 400 may be a section of a 360 degree panoramic imageor video. As depicted for ease of reference, image 400 is shown in a 2Dperspective. However, image 400 may be a 3D video and/or image.

FIGS. 4B-4F illustrate examples of image 400 displaying the interior ofan aircraft with various integrated overlays, in accordance with someembodiments. In some embodiments, overlays including parallax lines,such as station-lines, butt-lines, and water-lines, are layered onto theimage for scaling and/or measurement purposes. Such overlays may bestored along with the captured images in server 112 and/or database 116.In some embodiments, autofocus features of recording device 118, or amounted device 220, accounts for the distance within an image andcreates parallax lines as needed to maintain a default and/oruser-defined spacing. The parallax lines may be spaced evenly at apredetermined distance (e.g., 5″, 10″, 12″, 18″, etc.).

FIG. 4B depicts image 400 with station-lines 402 a-402 j runningparallel to the X-axis of an aircraft. Station-lines, such asstation-lines 402 a-402 j generally designate locations along the lengthof the aircraft, run from the front of the structure to the rear, andare annotated at structurally significant items such as ribs. Forexample, a pipe that began at station 22 and ran straight to station 500without any clamps or bends or holes would include only a reference tothe start and stop points. In some embodiments, the origin of thefuselage station (FS=0 or x=0) is placed at the nose tip or somedistance ahead of the nose (e.g., approximately 50 to 100 inches).Additionally, evenly spaced station-lines may provide perspective forhorizontal distances of the structure in the Y-axis. For example, aspreviously described, parallax station-lines 402 a-402 j may be evenlyspaced at a predetermined distance (e.g., 5″, 10″, 12″, 18″, etc.).

FIG. 4C depicts image 400 with butt-lines 404 a-404 f running parallelto the Y axis of an aircraft, Butt-lines, such as butt-lines 404 a-404 fgenerally measure left and right of the aircraft centerline. In someembodiments, the origin of the butt line (BL=0 or y=0) is located at theaircraft plane of symmetry. Additionally, evenly spaced butt-lines mayprovide perspective for vertical distances of the structure in theZ-axis. For example, as previously described, parallax butt-lines 404a-404 f may be evenly spaced at a predetermined distance (e.g., 5″, 10″,12″, 18″, etc.).

FIG. 4D depicts image 400 with water-lines 406 a-406 j running parallelto the Z-axis of an aircraft. Water-lines, such as water-lines 406 a-406j generally designate location of important points in the height of theaircraft, from ground up (e.g., floor, ceiling, etc.). Typically,water-line 0 is generally a bit below the aircraft or the bottom of thefuselage. In some embodiments, the origin of the water line (WL=0 orz=0) is placed at the nose tip, at the ground, or approximately 100 to200 inches below the nose tip. Additionally, evenly spaced water-linesmay provide perspective for longitudinal distances of the structure inthe X-axis. For example, as previously described, parallax water-lines406 a-406 j may be evenly spaced at a predetermined distance (e.g., 5″,10″, 12″, 18″, etc.).

A grid overlay may be created by using a laser projection device mountedon recording device 118 with an autofocus capability. Such laserprojection device may be a mounted device 210. When pointed at anobject, the laser projection device has the ability to create a grid ofdefined lines in any of the perpendicular X-, Y-, and/or Z-axes. In someembodiments, stereoscopic camera 200 may include components providingcapabilities of a laser projection device. In some embodiments, aparallax grid overlay comprises one or more layers, where each layerincludes a set of parallax lines. In some embodiments, a set of parallaxlines may include a group of one or more parallel lines. In someembodiments, a set of parallax lines may include a group of one or morestation-lines, butt-lines, and/or water-lines. The sets of parallaxlines may be stored as meta data attached to the video and/or imageformat. For example, four specific channels may be captured: 1.) videoand/or image, 2.) station-lines, 3.) butt-lines, and 4.) water-lines. Insome embodiments, the captured data may be aggregated into a single fileretrievable for viewing.

Each set of parallax lines may be created and stored on separate videolayers and/or levels so they may be activated (e.g. turned on/oft) asneeded. In various embodiments, one or more layers of parallax lines maybe activated and displayed over the video and/or image. For example,each of FIGS. 4B-4D may represent a set of parallax lines captured on achannel and stored as meta data associated with the file for image 400.The meta data corresponding to image 400 and parallax lines representedin FIGS. 4B-4D may be aggregated into a single retrievable file forviewing. For example, FIG. 4E depicts image 400 with all overlays, asdescribed in FIGS. 4B-4D, activated and superimposed. Such parallax lineoverlays may be used for scaling for virtual customer walkthroughs orsupporting engineering analysis, etc. Such calibrated parallax lines maygive an engineer the ability to quickly geometrically assess the areasfor design updates and changes. It may also allow an offsite engineer toaccurately measure features that exist on the aircraft without having tophysically measure it or searching for legacy drawings.

Another type of overlay that may be created and displayed over image 400is a thermal scan overlay. FIG. 4F depicts image 400 with an integratedthermal scan overlay. The shaded regions 408 a-408 f representvisualized thermal radiation of a thermogram overlayed upon image 400.The different shaded patterns represent different temperature ranges.Infrared radiation is emitted by all objects with a temperature aboveabsolute zero. The amount of radiation emitted by an object increaseswith temperature, allowing variations in temperature to be visualizedthrough thermography. In various embodiments, a thermal scan overlay maycomprise a thermogram video and/or image of such infrared radiationdetected and captured by an infrared vision device mounted tostereoscopic camera 200. For example, an infrared vision device may be afocal plane array (FPA) infrared camera capable of detecting radiationin the mid (3 to 5 μm) and long (7 to 14 μm) wave infrared bands,denoted as MWIR and LWIR, respectively, corresponding to two of the hightransmittance infrared windows. In other embodiments, the infraredvision device may be one of various other thermographic cameras orrecording devices.

In some embodiments, a thermogram video and/or image may be capturedthrough passive thermography, in which features of interest arenaturally at a higher or lower temperature than the background.Alternatively, and or additionally, a thermogram video and/or image maybe captured via active thermography in which an energy source isrequired to produce a thermal contrast between the feature of interestand the background. Active thermography may be necessary in instanceswhere the inspected parts are usually in equilibrium with thesurroundings.

Once captured, the thermogram video and/or image may then be stored as aseparate overlay layer to image 400 in server 112 and/or database 116,and be activated (e.g. turned on/off) as needed. Such thermal scanoverlays may be used for supporting various engineering analyses. Forexample, thermographic imaging is a non-destructive test method and canbe used to measure or observe inaccessible or hazardous areas. It canalso be used to detect objects in dark areas. An infrared image thatintegrates accurate temperature data may provide technicians orengineers with crucial information about the condition of all kinds ofequipment and structures. It can be used to find defects in shafts,pipes, and other metal or plastic parts. It is also capable of recordingmoving targets in real time and allows comparison of temperatures of alarge area. As a non-contact measurement that also makes invisible heatissues visible, thermal cameras let technicians, engineers, or otheroperators inspect production equipment more safely even at peakoperation. Along with troubleshooting, thermal scan overlays can alsohelp optimize the production process itself as well as monitor qualitycontrol.

In some embodiments, video and/or images may be captured and viewed inreal-time. In some embodiments, the overlays described above may also betransmitted in real-time to the viewer. In various embodiments, theimages and overlays, such as a thermal scan overlay, previouslydiscussed with reference to FIGS. 4A-4F may be utilized forpost-processing analytics. Such post-processing analytics may includeautomatically identifying foreign object damage (“FOD”), determiningassembly compliance, and/or analyzing thermal deviation. Thesepost-processing analytics are further explained below with reference tooperation 513 of method 500 described in FIGS. 5A-5B. In someembodiments, the system is configured to create analytic reports bycomparing captured images, such as image 300 or 400, against one or moreanalytic databases.

Examples of Generating Enhanced Stereographic Video of Aircraft BuildProcesses

FIGS. 5A-5B illustrate a process flowchart corresponding to method 500for generating enhanced stereographic video of aircraft build processes,in accordance with some embodiments. Method 500 may be implemented byvarious embodiments of system 100 described above. In some embodiments,method 500 may be implemented as a specific portion of process 800described below with reference to FIG. 8, such as at least operations808, 810, 812, and 814.

At operation 501, a plurality of stages of an aircraft build process arecaptured via a stereoscopic recording device. In some embodiments,images and/or video of each build process may be captured by recordingdevice 118 or stereoscopic camera 200. Such build processes may includethe operations described below with reference to FIG. 8. For example,operation 501 may be implemented to capture video and/or images of theinterior and/or exterior of an aircraft during various stages of systemintegration 810, in which various components and subassembliesmanufactured in block 808 are assembled together. In some embodiments,the various stages of system integration 810 may be divided intodifferent build days, such as build day-1 810-A, build day-2 810-B,build day-3 810-C, and so on, to build day-N 810-D. In some embodiments,operation 501 may be implemented to capture video and/or images on anincremental time scale, such as one or more build days during componentand subassembly manufacturing 808. In some embodiments operation 501 maybe implemented to capture video and/or images of the aircraft atparticular milestones which may represent results from one or more builddays 810-A to 810-N.

At operation 503, a stereographic library is built, which includesrepositories 505 of 3D video organized by tail number. Every aircraft isregistered with a unique identifying combination of letters and numberswhich must be displayed on the outside of the aircraft. Organizing thestereographic library by tail number may make it more convenient to findthe videos and images corresponding to the desired aircraft. In someembodiments, a customer's access to videos and images may be limited tothose corresponding to tail numbers of aircraft purchased by suchcustomer. In some embodiments, the repositories 505 of 3D videocorrespond to the plurality of stages of the aircraft build process. Aspreviously described, such stages of the aircraft build process may befurther described below with reference to FIG. 8.

At operation 507, an enhanced walkthrough video 509 of the aircraftbuild process is generated. In some embodiments, an enhanced walkthroughvideo 509 is generated for each stage of the build process. In someembodiments, an enhanced walkthrough video 509 may include a pluralityof stages of the build process. In some embodiments, the enhancedwalkthrough video 509 includes one or more of the following: a parallaxgrid overlay 515 integrated into the video, and a thermal scan overlayintegrated into the video.

The parallax grid overlay 515 may include a plurality of parallax linesdetermined automatically using autofocus. As previously described, alaser projection device may be mounted onto stereoscopic camera 200 as amounted device 210. The laser projection device may be configured withautofocus capabilities to create a grid of defined lines in any of theperpendicular axes (e.g., X-axis, Y-axis, and Z-axis). In someembodiments, the parallax lines may be any combination of one or more ofthe parallax lines previously described with reference to FIGS. 4B-4E,such as station-lines 402 a-402 j, butt-lines 404 a-404 f, water-lines406 a-406 j, etc. In some embodiments, each type of parallax line iscaptured on a specific channel. The plurality of parallax lines may beorganized as one or more sets 516 of parallax lines. Each set 516 ofparallax lines in the plurality of parallax lines may be stored as aseparate video layer 517 in the parallax grid overlay 515. The sets ofparallax lines may be stored as meta data attached to the video and/orimage format. In some embodiments, the captured data may be aggregatedinto a single file retrievable for viewing. In some embodiments, a set516 of parallax lines includes one or more of one type of parallaxlines. In some embodiments, a set 516 of parallax lines may include anycombination of one or more of the parallax lines previously described.

In some embodiments, the parallax grid is configured such that accuratereal-life measurements 519 for locations, spacing, and aircraftstructures can be extracted from the enhanced walkthrough video 509. Forexample, as previously described with reference to FIGS. 4B-4E,station-lines may be evenly spaced at some predetermined distance (e.g.,5″, 10″, 12″, 18″, etc.) and can provide perspective for horizontaldistances of the structure in the Y-axis. Similarly, evenly spacedbutt-lines may provide perspective for vertical distances of thestructure in the Z-axis. Additionally, evenly spaced water-lines mayprovide perspective for longitudinal distances of the structure in theX-axis.

As also previously described, a thermal scan overlay may be captured byan infrared vision device, such as a thermographic camera, mounted tostereoscopic camera 200 as a mounted device 210. The thermogram videoand/or images captured may be stored as one or more overlays that may bedisplayed over an image, such as image 400. In some embodiments, athermal scan overlay may reveal temperature variations so clearly thatthe underlying image 400 is not necessary for analysis. Thus, anengineer or other technician may only need to view the thermal scanoverlay. Other overlays may additionally, and/or alternatively, becreated by other mounted devices 210 that measure other informationcorresponding to the surroundings of stereoscopic camera 200.

In some embodiments, the enhanced walkthrough video 509 is configured toallow remote in-process initial inspections 511. This novel methodprovides a solution for a customer to review the product at any point inthe manufacturing build cycle (as described in FIG. 8). A customer mayuse a client device 102-108 (e.g. iPad, mobile device, laptop, or smartglasses etc.) and view remote walkthroughs of previously recordedproduct intervals. In some embodiments, the system may allow thecustomer to experience a real-time interactive session in a second mode.This may allow a customer to identify potential issues with customerrequirements and/or to change previously selected options before asignificant amount of time and cost is expended. Initial inspections 511may also be used by technicians or engineers for engineering analyses.The image data can be analyzed to identify deviations to customerrequirements, to identify thermal variations of the assembly, and toperform inspections of the assembly. For example, calibrated parallaxlines may give an engineer the ability to quickly measure andgeometrically assess the areas for design updates and changes. Asanother example, thermal scan overlays can assist in identifyinganomalies and other mechanical and/or electrical issues.

Furthermore, post-processing analytics may be used to automaticallyidentify potential issues during the build process. At operation 513,the enhanced walkthrough video 509 is analyzed using post-processinganalytics to identify anomalies and irregularities that occurred duringthe aircraft build process. In some embodiments, the post-processinganalytics includes analyzing patterns and shapes to detect foreignobject damage 521. Foreign object damage (“FOD”) is any damageattributed to a foreign object that can be expressed in physical oreconomic terms and may or may not degrade the products required safetyor performance characteristics. FOD may describe both the damage done toan aircraft by foreign objects, and the foreign objects themselves.Foreign objects may be any object that is not part of the aircraft orvehicle, such as various tools, cell phones, badges, food, etc. In someembodiments, an FOD Database is stored within server 112 and/or database116. The FOD Database may include a library of anomalous shapescorresponding to foreign objects commonly found in an aircraft duringvarious build processes, including cell-phone objects, badges, keys,tools, rivets, brackets, wire or hydraulic scraps, etc. In someembodiments, the library includes information relating to such foreignobjects, including shapes corresponding to various perspectives, sizing,coloration, etc. of the foreign objects. During post-processing, animage, such as image 400, may be scanned for such foreign objects bymatching anomalous shapes in the image with the information stored inthe FOD Database. For example, the system may identify a screw-driver ora wrench that was left on the aircraft. In some embodiments, the systemmay be configured to search for specific foreign object types.

In some embodiments, the post-processing analytics includes analyzingpatterns and shapes to determine assembly and sub-assembly compliance523. For each aircraft and build-stage there may be expected shapes forassemblies, sub-assemblies, and customer unique options. These shapesmay be derived from the computer-aided design and computer-aidedmanufacturing (CAD/CAM) system which maintains the “as-built”configuration of an aircraft. Such expected shapes may be stored in aStandard (expected) Shape Database within server 112 and/or database116. For example, a Standard Shape Database may include shapeinformation relating to seats, tables, lavatories, overheadcompartments, and other standard structures of an aircraft. Duringpost-processing, an image, such as image 400, may be scanned for suchstandard structures by matching the shapes in the image with the shapeinformation stored in the Standard Shape Database in order to confirmcorrect proper installation of the various structures. For example,assembly compliance may be executed to confirm that lavatories areinstalled correctly in the expected position.

In some embodiments, the post-processing analytics includes analyzingpatterns and shapes to determine thermal gradient compliance 525. Foreach aircraft and build-stage there may be expected temperaturegradients of various areas of the aircraft. Temperatures which deviatefrom the normal can indicate an electrical or mechanical problem. Suchexpected temperature gradient information may be stored in a Thermal(expected) Heat Database within server 112 and/or database 116. Duringpost-processing, a thermal scan overlay, such as that previouslydescribed in conjunction with FIG. 4F, may be searched and compared withthe Thermal Heat Database to identify temperature gradients that are notin compliance within thresholds of expected temperatures. For example,the thermal behavior of a cabin light from an image, such as image 400,with a thermal scan overlay may be checked against the Thermal HeatDatabase to validate that the thermal gradient temperature is consistentwith the expected thermal behavior of a cabin light.

FIG. 6 is a block diagram illustrating an example of a computer system600 capable of implementing various processes described in the presentdisclosure. The system 600 typically includes a power source 624; one ormore processing units (CPU's) 602 for executing modules, programs and/©rinstructions stored in memory 612 and thereby performing processingoperations; one or more network or other communications circuitry orinterfaces 620 for communicating with a network 622; controller 618; andone or more communication buses 614 for interconnecting thesecomponents. In some embodiments, network 622 may be a wireless and/orwired network, such as network 110 previously described in FIG. 1. Insome embodiments, network 622 can be another communication bus, theInternet, an Ethernet, an Intranet, other wide area networks, local areanetworks, and metropolitan area networks. Communication buses 614optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.System 600 optionally includes a user interface 604 comprising a displaydevice 606, a keyboard 608, and a mouse 610.

Memory 612 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM or other random access solid state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 612 may optionallyinclude one or more storage devices 616 remotely located from the CPU(s)602. In some embodiments, memory 116 may comprise one or more storagedevices 616.

Memory 612, or alternately the non-volatile memory device(s) withinmemory 612, comprises a non-transitory computer readable storage medium.In some embodiments, memory 612, or the computer readable storage mediumof memory 612 stores the following programs, modules and datastructures, or a subset thereof:

-   -   an operating system 640 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a file system 644 for storing various program files;    -   an image capture module 646 for receiving 3D stereographic video        and/or images, such as image 400, from a recording device 118,        such as stereographic camera 200, as described in operation 501;    -   an image storage module 648 for storing the captured video        and/or images in memory 612, storage device 616, and/or database        116, and for organizing the video and/or images as repositories        corresponding to tail number, as described in operation 503;    -   a parallax line overlay module 650 for receiving data        corresponding to projected lines from stereoscopic camera 200        and/or mounted device 210 comprising a laser projection device,        and storing the data as an overlay corresponding to the        associated image in memory 612, storage device 616, and/or        database 116, as described in operation 507;    -   a thermal scan overlay module 652 for receiving data        corresponding to thermographic imaging captured by mounted        device 210 comprising an infrared vision device, and storing the        data as an overlay corresponding to the associated image in        memory 612, storage device 616, and/or database 116, as        described in operation 507;    -   a post-processing module 654 for comparing shapes and patterns        stored in memory 612, storage device 616, and/or database 116        against the shapes and patterns in a captured image and/or        overlay in order to automatically identify foreign object damage        (“FOD”), determine assembly compliance, and/or analyze thermal        deviation as described in operation 513; and    -   local database information 656 comprising aircraft        identification information, operating parameters, measurements,        anomalous shapes corresponding to foreign objects, expected        shapes corresponding to proper components and/or structures,        temperature gradient information, and/or other manufacturing        information.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. One or more ofthe above identified modules may operate by retrieving input from one ormore client devices 102-108 and/or local storage 616 or other databaseson network 622, such as database 116. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules may be combined or otherwise re-arranged invarious embodiments. In some embodiments, memory 612 may store a subsetof the modules and data structures identified above. Furthermore, memory612 may store additional modules and data structures not describedabove.

Although FIG. 6 shows a “system for generating enhanced stereographicvideos of aircraft build processes,” FIG. 6 is intended more asfunctional description of the various features which may be present in aset of servers than as a structural schematic of the embodimentsdescribed herein. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated. For example, some items shown separately inFIG. 6 could be implemented on single servers and single items could beimplemented by one or more servers. The actual number of servers used toimplement a system for generating enhanced stereographic videos ofaircraft build processes and how features are allocated among them willvary from one implementation to another, and may depend in part on theamount of data traffic that the system must handle during peak usageperiods as well as during average usage periods.

Examples of Aircraft and Methods of Fabricating and Operating Aircraft

To better understand various aspects of implementation of the describedsystems and techniques, a brief description of an aircraft and aircraftwing is now presented. FIG. 7 is a schematic illustration of aircraft700, in accordance with some embodiments. As depicted in FIG. 7,aircraft 700 is defined by a longitudinal axis (X-axis), a lateral axis(Y-axis), and a vertical axis (Z-axis). In various embodiments, aircraft700 comprises airframe 750 with interior 770. Aircraft 700 includeswings 720 coupled to airframe 750. Aircraft 700 may also include engines730 supported by wings 720. In some embodiments, aircraft 700 furtherincludes a number of high-level inspection systems such as electricalinspection system 740 and environmental inspection system 760. In otherembodiments, any number of other inspection systems may be included.

Aircraft 700 shown in FIG. 7 is one example of a vehicle for which anenhanced stereographic video may be generated at various stages of thebuild process, such as by system 100 by implementation of method 500, inaccordance with illustrative embodiments. Although an aerospace exampleis shown, the principles disclosed herein may be applied to otherindustries, such as the automotive industry. Accordingly, in addition toaircraft 700, the principles disclosed herein may apply to othervehicles, e.g., land vehicles, marine vehicles, space vehicles, etc.

Examples of the present disclosure may be described in the context ofaircraft manufacturing and service method 800 as shown in FIG. 8 andaircraft 700 as shown in FIG. 7. During pre-production, illustrativemethod 800 may include specification and design (block 804) of aircraft700 and material procurement (block 806). During production, componentand subassembly manufacturing (block 808) and system integration (block810) of aircraft 700 may take place. In some embodiments, systemintegration (block 810) may comprise one or more designated build days,including build day-1 810-A, build day-2 810-B, build day-3 810-C, andup to build day-N 810-D, in some embodiments, component and subassemblymanufacturing (block 808) and system integration (block 810) may occurconcurrently. For example, as various components and/or subassembliescomplete manufacturing in block 808, they may be integrated into theaircraft at block 810 while other components and/or subassemblies arebeing manufactured in block 808. Described systems, methods, andassemblies formed by these methods, can be used in any of specificationand design (block 804) of aircraft 700, material procurement (block806), component and subassembly manufacturing (block 808), and/or systemintegration (block 810) of aircraft 700.

Thereafter, aircraft 700 may go through certification and delivery(block 812) to be placed in service (block 814). While in service,aircraft 700 may be scheduled for routine maintenance and service (block816). Routine maintenance and service may include modification,reconfiguration, refurbishment, etc. of one or more inspection systemsof aircraft 700. Described systems, methods, and assemblies formed bythese methods, can be used in any of certification and delivery (block812), service (block 814), and/or routine maintenance and service (block816).

Each of the processes of illustrative method 800 may be performed orcarried out by an inspection system integrator, a third party, and/or anoperator (e.g., a customer). For the purposes of this description, aninspection system integrator may include, without limitation, any numberof aircraft manufacturers and major-inspection system subcontractors; athird party may include, without limitation, any number of vendors,subcontractors, and suppliers; and an operator may be an airline,leasing company, military entity, service organization, and so on.

Apparatus(es) and method(s) shown or described herein may be employedduring any one or more of the stages of manufacturing and service method(illustrative method 800). For example, components or subassembliescorresponding to component and subassembly manufacturing (block 808) maybe fabricated or manufactured in a manner similar to components orsubassemblies produced while aircraft 700 is in service (block 814).Also, one or more examples of the apparatus(es), method(s), orcombination thereof may be utilized during production stages (block 808)and (block 810), for example, by substantially expediting assembly of orreducing the cost of aircraft 700. Similarly, one or more examples ofthe apparatus or method realizations, or a combination thereof, may beutilized, for example and without limitation, while aircraft 700 is inservice (block 814) and/or during maintenance and service (block 816).

CONCLUSION

Different examples of the apparatus(es) and method(s) disclosed hereininclude a variety of components, features, and functionalities. Itshould be understood that the various examples of the apparatus(es) andmethod(s) disclosed herein may include any of the components, features,and functionalities of any of the other examples of the apparatus(es)and method(s) disclosed herein in any combination, and all of suchpossibilities are intended to be within the spirit and scope of thepresent disclosure. Many modifications of examples set forth herein willcome to mind to one skilled in the art to which the present disclosurepertains having the benefit of the teachings presented in the foregoingdescriptions and the associated drawings.

Therefore, it is to be understood that the present disclosure is not tobe limited to the specific examples illustrated and that modificationsand other examples are intended to be included within the scope of theappended claims. Moreover, although the foregoing description and theassociated drawings describe examples of the present disclosure in thecontext of certain illustrative combinations of elements and/orfunctions, it should be appreciated that different combinations ofelements and/or functions may be provided by alternative implementationswithout departing from the scope of the appended claims. Accordingly,parenthetical reference numerals in the appended claims are presentedfor illustrative purposes only and are not intended to limit the scopeof the claimed subject matter to the specific examples provided in thepresent disclosure.

What is claimed is:
 1. A system, comprising: one or more processors;memory; and one or more programs stored in the memory, the one or moreprograms comprising instructions for: building a stereographic libraryincluding repositories of 3D video, the repositories of 3D videocorresponding to a plurality of stages of an aircraft process, whereinthe repositories of 3D video include integrated overlays associated withthe 3D video, the integrated overlays comprising thermal scans and setsof parallax grids including a plurality of parallax lines; generating anenhanced walkthrough video of the aircraft build process, the enhancedwalkthrough video including one or more of the integrated overlayswherein each integrated overlay included in the enhanced walkthroughvideo can be activated or deactivated as needed; and analyzing theenhanced walkthrough video using post-processing analytics.
 2. Thesystem of claim 1, wherein the post-processing analytics includesanalyzing at least one of a pattern or shape to detect foreign objectdamage.
 3. The system of claim 1, wherein the post-processing analyticsincludes analyzing at least one of a pattern or shape to determineassembly and sub-assembly compliance.
 4. The system of claim 1, whereinthe post-processing analytics includes analyzing patterns and shapes todetermine thermal gradient compliance.
 5. The system of claim 4, whereindetermining thermal gradient compliance includes matching thermal heatto a thermal heat database storing expected temperature gradientinformation therein.
 6. The system of claim 1, wherein the thermal scansare created using passive thermography.
 7. The system of claim 1,wherein the thermal scans are created using active thermography.
 8. Thesystem of claim 1, wherein the thermal scans are created using aninfrared vision device.
 9. The system of claim 8, wherein the infraredvision device is mounted on a stereoscopic camera.
 10. The system ofclaim 9, wherein the infrared vision device is a focal plane arrayinfrared camera capable of capturing two different high transmittanceinfrared windows.
 11. A method comprising: building a stereographiclibrary including repositories of 3D video, the repositories of 3D videocorresponding to a plurality of stages of an aircraft build process,wherein the repositories of 3D video include integrated overlaysassociated with the 3D video, the integrated overlays comprising thermalscans and sets of parallax grids including a plurality of parallaxlines; generating an enhanced walkthrough video of the aircraft buildprocess, the enhanced walkthrough video including one or more of theintegrated overlays, wherein each integrated overlay included in theenhanced walkthrough video can be activated or deactivated as needed;and analyzing the enhanced walkthrough video using post-processinganalytics.
 12. The method of claim 11, wherein the post-processinganalytics includes analyzing patterns and shapes to detect foreignobject damage.
 13. The method of claim 11, wherein the post-processinganalytics includes analyzing patterns and shapes to determine assemblyand sub-assembly compliance.
 14. The method of claim 11, wherein thepost-processing analytics includes analyzing patterns and shapes todetermine thermal gradient compliance.
 15. The method of claim 14,wherein determining thermal gradient compliance includes matchingthermal heat to a thermal heat database storing expected temperaturegradient information therein.
 16. The method of claim 11, wherein thethermal scans are created using passive thermography.
 17. The method ofclaim 11, wherein the thermal scans are created using activethermography.
 18. The method of claim 11, wherein the thermal scans arecreated using an infrared vision device.
 19. A non-transitory computerreadable storage medium storing one or more programs configured forexecution by a computer, the one or more programs comprisinginstructions for: building a stereographic library includingrepositories of 3D video, the repositories of 3D video corresponding toa plurality of stages of an aircraft build process, wherein therepositories of 3D video include integrated overlays associated with the3D video, the integrated overlays comprising thermal scans and sets ofparallax grids including a plurality of parallax lines; generating anenhanced walkthrough video of the aircraft build process, the enhancedwalkthrough video including one or more of the integrated overlays,wherein each integrated overlay included in the enhanced walkthroughvideo can be activated or deactivated as needed; and analyzing theenhanced walkthrough video using post-processing analytics.
 20. Thenon-transitory computer readable storage medium of claim 19, wherein thepost-processing analytics includes analyzing patterns and shapes todetermine thermal gradient compliance including matching thermal heat toa thermal heat database storing expected temperature gradientinformation therein.